Stop the asset page's KPIs counting a day the chart does not show - #2434
Conversation
Context: - getAssetKPIs advanced its end date by a day to "make the end date inclusive", but both callers already hand it the chart's exclusive end, and the endpoint ends its own window before `end` as well (_get_sensor_stats filters event_start < end). - Every KPI therefore covered one day more than the chart beside it. On a seeded asset with one unit of energy per day, a three-day selection reported a total of four. - Advancing the date also mutated it in place, and that Date object is the one held by storeEndDate and previousResult.end, so the chart's own end date moved a day forward on every selection. Not advancing it at all fixes both. Change: - Pass the chart's end date through unchanged. - Assert the endpoint's window semantics, which the page depends on. Signed-off-by: F.N. Claessen <felix@seita.nl>
Context: - The entry was added before the PR existed, with an XXXX placeholder. Change: - Point it at PR #2434. Signed-off-by: F.N. Claessen <felix@seita.nl>
Context: - The KPI day-counting fix was split out into PR #2434, since it changes user-visible numbers while this PR is a performance change. Both branches carry the same code change and merge cleanly either way, but the changelog would have gained the line twice. Change: - Drop the entry here. The date-corruption entry stays, as that fix is needed by the cache in this PR. Signed-off-by: F.N. Claessen <felix@seita.nl>
There was a problem hiding this comment.
Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.
Note
This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.
Documentation build overview
11 files changed ·
|
Context: - The advance was not arbitrary. Until PR #1909 this function read picker.getEndDate(), which is the last day selected, so advancing it by a day correctly produced an exclusive end. - PR #1909 (v0.30.3, January 2026) gave the function start and end parameters and had the callers pass the chart's already-exclusive end, but kept the advance, which from then on added a day too many. Change: - Note that history where the advance used to be, so it does not get put back. Signed-off-by: F.N. Claessen <felix@seita.nl>
There was a problem hiding this comment.
Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.
Note
This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.
There was a problem hiding this comment.
Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.
Note
This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.
There was a problem hiding this comment.
Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.
Note
This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.
|
Copilot review requested three times (17:22, 17:31, 17:35 UTC); each attempt came back with "Copilot encountered an error and was unable to review this pull request", referring to runner configuration. Requests on #2433 and #2435 in the same window succeeded, including ones touching this same file, so it looks specific to this PR or transient on their side rather than something in the diff. Not retrying further to avoid noise. @Flix6x, worth a re-request from your side later, or a human review — the change is one line plus a test. CI is green: 13/13 checks pass. |
Context: - toIsoStringWithOffset appended the local UTC offset to date.toISOString(), which is UTC, without moving the clock time, so the string named an instant wrong by exactly that offset. - getAssetKPIs is its only caller, so this shifted the very window this PR is about. West of UTC the shift lands on a different day for sensors of daily resolution, so KPIs could report a different day than the chart. Change: - Write the local clock time, then append the offset. - The same fix is in PR #2435, where the JavaScript tests that found it live. Both branches carry identical text, so they merge either way. The changelog entry for it stays in #2435. Signed-off-by: F.N. Claessen <felix@seita.nl>
|
Added the The file is byte-identical to the version in #2435, where the JavaScript tests that found it live, so the branches merge in either order. The changelog entry for it stays in #2435, to avoid the line appearing twice. |
There was a problem hiding this comment.
Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.
Note
This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.
|
Update on the Copilot review: five attempts now, across three different commits, spanning about two hours — every one returned "Copilot encountered an error and was unable to review this pull request".
Requests on #2433 and #2435 succeeded during the same windows, including ones touching this same file, so it is specific to this PR rather than a general outage or anything in the diff. I have stopped retrying. This PR therefore has had no automated review at all, which is worth knowing given it is the one that changes reported KPI values. @Flix6x, a re-request from your account may fare better; otherwise it is worth a human read. The change is two one-line fixes plus a test. CI is green: 13/13. |
|
Copilot could not review this one (five attempts, all erroring), so I ran three adversarial reviewers locally instead — one on the KPI-window claim, one on timezone correctness, one on test quality. Between them they found five things worth acting on. All are now addressed; the core claim survived. What survived. The claim that both callers pass an exclusive end was checked by rendering the page for What changed.
One thing I am leaving, and flagging instead. Removing the advance also stops 99 tests pass across the two touched modules, under random ordering. |
Context: - Review by @nhoening: PR #2434 now isolates the KPI window fix, so this PR should carry neither the changelog entry nor the code, nor its test. - Almost right. What this PR genuinely needs is narrower than the fix: the advance must not reach storeEndDate or previousResult.end, because both are handed to the cache, which would then record a span a day wider than it holds. It does not need the advance gone. Change: - Restore getAssetKPIs to what main has, advance and all, so the fix and its user-visible effect belong wholly to PR #2434. - Pass copies at both call sites, which changes no KPI value and keeps the chart's own dates out of reach. Once #2434 lands these are belt and braces, and the two branches merge either way. - Drop the changelog entry and test_kpi_window_end_is_exclusive, which #2434 carries. - Say why replay fetches around the cache rather than through it. Signed-off-by: F.N. Claessen <felix@seita.nl>
…2433) * ui/graphs: stop the KPI fetch from corrupting the chart's date range Context: - getAssetKPIs() advances the end date it is given by one day, to make the KPI window inclusive. - On the asset page, it was handed the very Date objects that storeEndDate and previousResult.end hold, so after one date selection both were a day late. The exact-range short-circuit in fetchGraphDataAndKPIs then stopped matching, and embedAndLoad sliced a day too much off the cached data. - The initial call additionally read storeStartDate/storeEndDate from the module body, which is evaluated before the DOMContentLoaded handler that assigns them (verified in headless Chrome), so on an asset with KPIs it threw on undefined and aborted the rest of the module. Change: - Pass copies to getAssetKPIs at both call sites. - Derive the initial window from the template values rather than from the not-yet-assigned globals. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: fetch chart data through a single module Context: - Four call sites in graphs.html (initial load, sensors-to-show reload, replay, and date selection) each built the same /chart_data query string by hand and repeated the same response handling. - That duplication is what makes a reuse cache awkward to add: it would have to be threaded through every call site separately. Change: - Add chart-data-source.js with buildChartDataUrl, fetchChartData and fetchChartAnnotations, and route all five fetches through it. - Drop the now-unused decompressChartData import, the dead module-level queryStartDate/queryEndDate, and two parameters of fetchGraphDataAndKPIs that the caller no longer needs. - No behaviour change: the generated URLs are unchanged. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: fetch only the newly selected days, not the whole window Context: - Issue #101: changing the selected time window re-queried the entire window, even when almost all of it was already loaded. Only an exactly identical selection was served from memory. - Profiling a 5-sensor asset holding a year of 15-minute data: a request costs ~105 ms regardless of size, plus ~24 us per record. Re-fetching a year takes ~4.4 s and 14 MiB, of which ~93% is server time. Change: - Add chart-data-cache.js, which works out what a newly selected window adds to the loaded one and fetches only that. - The date picker only yields whole nominal days, so the loaded window is always one contiguous range; interval arithmetic stays on Date objects so a 23- or 25-hour day across a DST transition stays correct. - Merged records are de-duplicated, since an event straddling the seam is returned by both halves. - A window fully inside the loaded one now needs no request at all. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/tests: pin down that a chart data window can be fetched in parts Context: - The UI now fetches only the part of a newly selected window that it does not already hold, which assumes that [a, b) plus [b, c) equals [a, c). Change: - Assert that splitting a window loses no events and changes no values, for resolutions that divide a day and for one (7 minutes) that does not. - Assert that the event straddling the seam is repeated rather than dropped, and that both halves agree on its value, which is what makes de-duplicating in the front end sufficient. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: changelog entries for the chart data reuse and the KPI date fix Context: - Both changes are user-visible: charts respond faster to a new date selection, and the KPI-related date corruption is a bugfix. Change: - Add one entry under New features and one under Bugfixes for v1.1.0. - PR numbers are placeholders (XXXX) until the PR exists. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: keep the widest span loaded, so zooming back out is free Context: - The reuse introduced in the previous commit trimmed its records to the window on display, so narrowing the selection discarded everything outside it. Zooming into 7 days of a loaded 31-day window and back out then re-fetched the other 24 days in two requests. Change: - Hold the widest contiguous span fetched so far and clip only what is handed to the chart, so a narrowed selection and a return to the earlier span both cost nothing. - The cache is bounded by the widest contiguous span browsed, which is what selecting that span in one go would have loaded anyway. Selecting a window that does not touch the cached span replaces it, so the cached span stays contiguous. - Reset the cache in reloadChartData, where the sensors shown or the data itself may have changed and nothing held is still trustworthy. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: fill in the PR number in the changelog entries Context: - The entries were added before the PR existed, with XXXX placeholders. Change: - Point both entries at PR #2433. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: rename chart-data-source.js to chart-data-fetch.js Context: - Review feedback: "source" is reserved for belief sourcing in this repo, so naming a module after it is misleading. Change: - Rename the module and update its two importers. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: reuse only what a direct fetch would have returned Context: - Review feedback: clipping cached records by event_start does not match the API, which selects events *overlapping* the window (event_ends_after / event_starts_before). A 50-minute sensor lost its leading event whenever the window was narrowed, since that event starts before midnight and runs into the day. - Investigating that surfaced a second, larger mismatch: the API anchors its resampling at the start of the window asked for, so a coarser sensor shown next to a 7-minute one comes back on shifted timestamps for a window offset by a fraction of the resolution. Those records cannot be reconciled with the ones already held. Change: - Clip on overlap rather than on event_start, and match instantaneous events inclusively on both edges, as the API does. - Derive the resolution the events are actually spaced on: every sensor with a resolution is resampled to the finest one requested, while the response still reports each sensor's own. - Refuse to reuse records for a window that is not on the same resampling grid, falling back to fetching the window whole. Whole-day selections of sensors whose resolution divides a day are always aligned, so this only bites the cases that were wrong before. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: seed the cache with the initially loaded window Context: - Review feedback: the cache was only ever filled by a date selection, so the first selection after a page load found it empty and re-fetched the whole newly selected range, even though the initial window was already in memory. The same applied after a reload triggered by a data change. Change: - Route the initial load and the post-reset reload through the cache, so the optimisation applies from the first date change onwards. Neither changes what is requested: an empty cache fetches the whole window. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/tests: pin down the window semantics the chart cache relies on Context: - The front end reproduces a fetch from records it already holds, so it depends on three behaviours that were not covered. Change: - Assert that a window selects events overlapping it, not only events starting inside it. - Assert that instantaneous events are included on both window edges. - Assert that resampling is anchored at the requested window start, which is why the front end refuses to reuse records across such a shift. - Reflow two docstrings to break only after punctuation, per the repo convention. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: treat touching windows as one contiguous span Context: - Review feedback: reuse required a strict overlap, so a selection stepped on by exactly its own width (start === cached.end) counted as disjoint and replaced the cache. Stepping back then re-fetched a window that had just been thrown away, which is the opposite of holding the widest span. Change: - Allow the boundaries to touch, so such a selection extends the cached span instead of replacing it. What gets fetched is unchanged: the new window is still entirely missing and is still fetched whole. - Note in missingRanges that the half-open convention has one exception, instantaneous sensors, which the API matches inclusively on both edges. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: stop getAssetKPIs from mutating the window it is given Context: - Review feedback: passing defensive copies at both call sites left the trap in place, since the next caller has to know to do the same. That is what caused the date corruption fixed in 04d6b1b. Change: - Advance a copy of the end date inside getAssetKPIs, and drop the copies at the call sites, which can now pass the chart's own window dates. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: break the JavaScript comments only after punctuation Context: - The repo's line-break convention was written for Python, but it exists for stable review comments and text search, which applies just as much to the JavaScript comments added here. Change: - Reflow the JSDoc blocks and inline comments in chart-data-cache.js, chart-data-fetch.js and the graphs.html module, so that every physical line ends at a comma, semicolon, colon or period. Signed-off-by: F.N. Claessen <felix@seita.nl> * agents/docstrings: extend the line-break convention to JavaScript Context: - The convention only claimed to apply to Python, so JavaScript comments were outside it, even though the reasons for it (stable review comments and text search) do not depend on the language. Change: - Widen applyTo to .js and .html, retitle to "Docstrings and comments", and scope the RST/Click/doctest guidance as Python-specific. - Add a JavaScript section covering JSDoc blocks and // comments, in .js files and in the script blocks of Jinja templates, with an example, and note that @param/@returns lines stay on one line however long. - State that automated reviewers should report JavaScript comments that wrap mid-phrase, as they already do for Python docstrings. Signed-off-by: F.N. Claessen <felix@seita.nl> * agents/docstrings: match how the other instruction files scope themselves Context: - Review feedback: the comma-separated applyTo glob was unlike any other instruction file here, and a consumer reading it as one literal pattern would match nothing, silently disabling the instructions altogether. Change: - Use applyTo "**", as the other cross-language instruction files do. The headings already separate the Python-specific guidance from the JavaScript section, so the scoping does not need to live in the glob. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: require the options that the fetch helpers cannot work without Context: - Review feedback: the helpers defaulted their options object to {} and then dereferenced start and end regardless, so the default only served to postpone the failure and suggest the argument was optional. Change: - Drop the defaults, since every one of these helpers needs a window. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: stop the KPIs counting a day the chart does not show Context: - Review feedback: getAssetKPIs advanced its end date by a day to "make the end date inclusive", but both callers already hand it the chart's exclusive end, and the endpoint ends its own window before `end` too (_get_sensor_stats filters event_start < end). - Every KPI therefore covered one day more than the chart beside it. On a seeded asset with one unit of energy per day, a three-day selection reported a total of four. Change: - Pass the chart's end date through unchanged. Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0/tests: pin down that the KPI window's end is exclusive Context: - The asset page derives the KPI window from the chart's, so the two have to agree on whether the end is part of the window. Nothing said so. Change: - Assert that a three-day window over one daily value per day totals three, not four. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: changelog entry for the KPI window fix Context: - KPI figures change for anyone using them, so this needs saying. Change: - Add a Bugfixes entry for v1.1.0. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: correct why the initial window is derived from the template Context: - Review feedback: the comment claimed this code runs in the module body, before the DOMContentLoaded handler assigns storeStartDate/storeEndDate. That is wrong. Checked against the parsed page: on the asset page the block sits inside that handler, so the globals are already assigned. - The real reason not to read them is worse. The handler's closing brace sits inside a Jinja conditional, the sensor upload block, so the handler ends at that point only when it renders. On every other page it runs on to the end of the module. Whether this code is inside the handler therefore depends on the page, and deriving the window from the template values is what works either way. - Consequently the claim that the chart failed to load on an asset page's first visit was wrong too: the globals were assigned by then. The date corruption that 04d6b1b also fixed was real and is unaffected. Change: - Replace the comment with the actual reason. - Drop the incorrect half of the changelog entry. - Tie the KPI window test to the requesting user's account, rather than to whichever account the database returns first. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: read the initial window from the globals already assigned Context: - Twice now I claimed this block was not inside the DOMContentLoaded handler: first that it ran in the module body, then that the handler's extent varied per page because its closing brace sat in a Jinja conditional. Both were inferred from indentation and both were wrong. - Parsing the rendered pages settles it. The handler spans the same region on the asset page and on the sensor page, closing unconditionally at the end of the timezone check, and the initial window is inside it and after the assignment of storeStartDate/storeEndDate in every case. - The brace that misled me closes the sensor upload click listener. It is written at four spaces where the listener is opened at eight, so it reads as the end of the handler. Change: - Use storeStartDate/storeEndDate, as main did, dropping the second copy of the window that was only introduced to work around a problem that does not exist. - Indent the upload listener's closing brace to match its opening, so the next reader is not misled the same way. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: leave the KPI window entry to its own PR Context: - The KPI day-counting fix was split out into PR #2434, since it changes user-visible numbers while this PR is a performance change. Both branches carry the same code change and merge cleanly either way, but the changelog would have gained the line twice. Change: - Drop the entry here. The date-corruption entry stays, as that fix is needed by the cache in this PR. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: keep the cache consistent when selections overlap Context: - Date selections are not serialised and their requests are not aborted, so two can be in flight at once. The merge read what was held *after* awaiting, by which time another selection may have replaced it. - Selecting [1, 7) and then [20, 25), with the second answered first, left the cache claiming [1, 25) while holding only the two ends. Selecting [10, 12) afterwards was then served from that gap: an empty chart, with no request made. The widening selection itself also returned 2 of its 6 days, since it merged against the unrelated window. Change: - Snapshot what is held on entry and merge against that snapshot, so a result's span and its records always describe the same thing. - Return the window this call assembled, rather than whatever is held by the time it finishes. Whichever selection finishes last decides what is kept; both spans are self-consistent, so the worst case is a later selection re-fetching. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: ignore the answer to a window no longer on screen Context: - Review feedback: the picker's selected handler starts its requests without aborting any still in flight, and whichever settles last draws itself and becomes previousResult. A slow earlier selection answering after a quick later one therefore put the older range on screen, while the picker and the URL named the newer one. - The cache fix in 9928c43 addressed a different race, inside the cache. It kept what is held self-consistent, but said nothing about which answer the page draws. Change: - Count how often the shown window changes, claim that count when a selection starts, and drop the result if the count has moved on. - Bump the same count when reloadChartData runs, since a reload for changed data supersedes any selection still in flight. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: guard the reload against being overtaken too Context: - Review feedback: 1b95e9c had reloadChartData bump the shown-window count so that a selection in flight could not overwrite it, but the reload never checked the count itself. A range selected while a reload was running could therefore render first, and the reload could then finish and draw the window the user had already moved away from. - The guard was written in one direction only. Change: - Have the reload claim a count as the selection handler does, and drop its result if a window has been shown since. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: give the date picker the range it is meant to open on Context: - The initial load declared `var sessionStart` and `var sessionEnd` inside the DOMContentLoaded handler, far below the picker's creation. Being `var`, they were hoisted to the top of that handler and shadowed the module-level values assigned earlier. - So createPicker(sessionStart, sessionEnd, ...) and, on simulation servers, computeSimulationRanges(...) both received `undefined` rather than the session's window. Confirmed by parsing the rendered page, which puts the declaration at line 341 and the use at line 217 within one handler spanning 170 to 434, and by running the same shape in a browser. Change: - Drop the two shadowing declarations and build the dates where they are used, so the module-level values reach the picker. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: guard the first load against being overtaken as well Context: - Review feedback: the picker is created before the initial load starts, and that load can take seconds on a wide window, so a range selected while it runs would render first and then be overwritten when the first load resolved. - That is the third async path onto the chart, after the selection handler and the reload, and the only one still unguarded. Change: - Claim a count for the initial load and draw its result only if no window has been shown since. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: reference only the PR in the changelog entry Context: - Review feedback: the entry linked both the PR and the issue it closes. Change: - Drop the issue link. The other entries on this branch, and those on PR #2434 and PR #2435, already reference only their PR. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: drop the last comment claiming getAssetKPIs mutates its argument Context: - Found while adversarially reviewing PR #2434: this branch corrected the inline comment about the fastChartWindow snapshot but left the block comment above the declaration still giving the old reason, that getAssetKPIs bumps storeEndDate by a day. It does not, since bc6d5c1. Change: - Give the snapshot its real reason, matching the wording on PR #2434 so the two branches stay identical here. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: hand the KPI window fix to its own PR Context: - Review by @nhoening: PR #2434 now isolates the KPI window fix, so this PR should carry neither the changelog entry nor the code, nor its test. - Almost right. What this PR genuinely needs is narrower than the fix: the advance must not reach storeEndDate or previousResult.end, because both are handed to the cache, which would then record a span a day wider than it holds. It does not need the advance gone. Change: - Restore getAssetKPIs to what main has, advance and all, so the fix and its user-visible effect belong wholly to PR #2434. - Pass copies at both call sites, which changes no KPI value and keeps the chart's own dates out of reach. Once #2434 lands these are belt and braces, and the two branches merge either way. - Drop the changelog entry and test_kpi_window_end_is_exclusive, which #2434 carries. - Say why replay fetches around the cache rather than through it. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/graphs: stop reusing records indefinitely Context: - Review by @nhoening: the cache had no age limit. Data written elsewhere, by a report or another user, raises no event in this tab, so a window left open kept answering from what was true when it was opened. Changing the date range used to fetch, and so used to show such data; with the cache it no longer did. Change: - Hold records for five minutes. Long enough that stepping or widening a selection is still answered from memory, short enough that a page left open picks up data written elsewhere before long. - Extending a span keeps its original age, so it cannot be kept alive indefinitely by being added to. - The limit is a parameter of the cache, so tests can age it in milliseconds rather than waiting. Signed-off-by: F.N. Claessen <felix@seita.nl> --------- Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: Nicolas Höning <nicolas@seita.nl> Co-authored-by: Nicolas Höning <nicolas@seita.nl>
* ui/tests: run the JavaScript modules in a browser, from pytest Context: - The modules under ui/static/js had no tests, and the only way we had checked them was by hand. They are plain ES modules, so they can be exercised directly, but ES imports do not work over file://. - A Node.js toolchain would be a heavy addition for ~5k lines of JavaScript, and would need its own CI job. Change: - Add a fixture that serves the modules over HTTP and runs a page of assertions in headless Chrome, reporting the results through pytest. Tests pass a snippet of JavaScript and assert on the checks it records. - Allow a test to place the browser in a chosen timezone, so that behaviour around daylight saving is not tied to the machine. - Add selenium to the test dependency group, which the runtime image does not install. The tests skip when selenium or Chrome is missing. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/tests: cover the date, data and chart-data helpers Change: - Test toIsoStringWithOffset, subtract and countDSTTransitions, the last under two fixed timezones, one with daylight saving and one without. - Test getUniqueValues and convertToCSV. - Test decompressChartData, including the pass-through of data already in the old format and the conversion of seconds-valued sensors to dates. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/static: fix two bugs the new JavaScript tests found Context: - toIsoStringWithOffset appended the local UTC offset to date.toISOString(), which is UTC, without moving the clock time. The result named an instant wrong by exactly that offset. Its only caller sends the KPI window to the API, so KPIs could cover the wrong days, by a whole day west of UTC. - getUniqueValues ended its loop at the first falsy entry, and tested for values already seen with `in`, which consults the prototype chain, so a source named "constructor" or "toString" was silently dropped. Change: - Write the local clock time, then append the offset. - Iterate by index, track what has been seen in a Set, and guard the nested lookup against a missing record. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: changelog entries for the JavaScript tests and the KPI time range fix Context: - One is infrastructure for plugin developers and hosts, the other is a user-visible fix. Change: - Add both to v1.1.0. PR numbers are placeholders until the PR exists. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: fill in the PR number in the changelog entries Context: - The entries were added before the PR existed, with XXXX placeholders. Change: - Point both at PR #2435. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/static: stop counting a record with no source as a source Context: - Review feedback on getUniqueValues: a record that lacks the key yields undefined, which then landed in the results. - Its only caller, checkSourceMasking, counts distinct source ids to decide whether to warn that data is being masked, so one record without a source made it warn about masking that was not happening. - This predates the null-entry fix in this PR: a record carrying no source at all already contributed an undefined. Change: - Skip values that are absent, and assert that a record without a source contributes nothing. Signed-off-by: F.N. Claessen <felix@seita.nl> * build: lock the selenium test dependency Context: - CI installs from uv.lock with --frozen, so adding selenium to the test group without relocking failed the pre-commit check and the image build. - Locking with an older uv rewrote the file to an earlier revision, so this was done with 0.10.9, the version CI pins. Change: - Add selenium and its eight dependencies. Nothing else moved: no package was removed and no existing version changed. - Apply black to the new test fixture. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/tests: format with the black version this repo pins Context: - CI failed on formatting the new test files. The repo pins black 26.3.1, which hugs multiline strings; the black in my environment was 25.1.0, which leaves them wrapped, so it reported the files as clean. Change: - Reformat with 26.3.1. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/tests: break the run() docstring only after punctuation Context: - Review feedback: the sentence wrapped after "daylight saving", against the repo convention, which PR #2433 has just extended to cover JavaScript as well as Python. Change: - Reflow so each line ends at a comma or a period. Signed-off-by: F.N. Claessen <felix@seita.nl> * ui/static: say in the JSDoc that the clock time written is the local one Context: - Adversarial review of PR #2434: the example did not say which clock the components come from, which is exactly what the fix changed. Change: - State it, and keep the file identical to the copy on PR #2434. Signed-off-by: F.N. Claessen <felix@seita.nl> --------- Signed-off-by: F.N. Claessen <felix@seita.nl>
There was a problem hiding this comment.
I believe the main fix is a good idea and I like one half of the tests that are being added. So the changes I ask for a overseeable.
However, I ran into a different problem while testing, which I believe might be good to fix here, as well - if we can identify the underlying problem.
My manual tests do not seem to get the right results yet, for multiple selected days.
The KPI is measuring energy costs, so it is summing up data (entry in sensors_to_show_as_kpis: {"title": "Daily costs", "sensor": 49173, "function": "sum"})
I had test data with ~122 EUR energy costs for Jan 15, 2030. I added 100 EUR for Jan 16, 2030 (note: different data source, as I uploaded one data point).
Selecting one day works well:
- Jan 14: KPI widget shows 0 EUR
- Jan 15: KPI widget shows 122 EUR
- Jan 16: KPI widget shows 100 EUR
Here is a typical API query: http://localhost:5000/api/v3_0/assets/4497/chart_data?event_starts_after=2030-01-15T23:00:00.000Z&event_ends_before=2030-01-16T23:00:00.000Z&compress_json=true
Now I select multiple days:
- Jan 14 + Jan 15: Total costs 122 EUR (GOOD)
- Jan 15 + Jan 16: Total costs 100 EUR (FALSE, I expected 222 EUR ... API query)
In fact, all day ranges including Jan 16 give me 100 EUR...
Maybe this is an unrelated bug, as the behavior shows up on main, too (but with the original bug of selecting one day more). I also checked on #2435 as that has a fix for an offset issue, but observed the same outcome.
| The values differ per day, so that a window covering the wrong days totals differently, | ||
| rather than merely covering the same number of days. | ||
| """ | ||
| from datetime import datetime |
There was a problem hiding this comment.
This import and the next should go to the top of the module.
There was a problem hiding this comment.
Done — datetime, pytz.utc, GenericAssetType and TimedBelief are now imported at the top of the module, and the in-function imports are gone.
| return asset, window_start | ||
|
|
||
|
|
||
| def _kpi_total(client, asset, start, end): |
There was a problem hiding this comment.
This could use type hinting, certainly for the return value.
Same for the utility above.
There was a problem hiding this comment.
Done. _asset_with_daily_kpi returns tuple[GenericAsset, datetime] and _kpi_total takes (client, asset: GenericAsset, start: str, end: str) -> float.
There was a problem hiding this comment.
I'm not really convinced by these tests. They are testing some code artefacts, e.g. not using setDate() which has gone as of this PR, or using another function twice.
We should test outcomes if anything.
I do like the API tests added in this PR, and the JS code is basically wrapping an API call. IMO this test here can go.
There was a problem hiding this comment.
Agreed, and removed. You are right that asserting on setDate being absent tests how the page is written rather than what it does, and it would have gone stale on its own.
What it was guarding — that getAssetKPIs does not move the window it is given — is now covered by outcome instead: test_kpi_window_end_is_exclusive and test_kpi_window_honours_the_offset_it_is_given both fail if the window sent is wrong, and the new test_kpi_reports_what_the_chart_draws asserts the reported total equals what the chart draws. I checked each goes red for the right reason before keeping it.
Signed-off-by: Nicolas Höning <nicolas@seita.nl>
Kept both changelog entries: PR #2435 merged its own, this branch keeps its. Signed-off-by: F.N. Claessen <felix@seita.nl>
Context: - @nhoening found that multiple selected days gave the wrong total: with January 15 worth 122 from one source and January 16 worth 100 from an uploaded point, the two days together reported 100 rather than 222, and every range including the 16th reported 100. - The cause is that the KPI read get_sensor_stats, which groups belief rows by data source, and then took whichever source the database returned first. Reproduced exactly, and on this machine the arbitrary winner was the other one, which is itself the point. - The same reading has a second consequence: those stats aggregate every belief row, with no belief-time filter, so a revised value was added to the value it revised. One day believed first as 50 and later as 122 reported 172, where the chart shows 122. Change: - Read the beliefs the chart draws, one value per event, and reduce those. A KPI is read beside the chart, so it should total what the chart totals. - get_downsample_function_and_value now takes those values rather than per-source statistics. Its only caller is this endpoint; the sensor stats endpoint is untouched, since a per-source breakdown is the point there. Signed-off-by: F.N. Claessen <felix@seita.nl>
Context: - Review by @nhoening: the rendered-page assertions test artefacts rather than outcomes, the imports belong at the top of the module, and the helpers should carry type hints. Change: - Drop the rendered-page test. What it guarded, that getAssetKPIs does not move its window, is now covered by asserting the reported total. - Add a test that a KPI totals exactly what the chart draws, over a sensor with two sources and a revised belief. It fails on the previous behaviour, reporting 100 or 172 where the chart shows 229. - Hoist the imports, type the helpers, and correct the expectation of the offset test: now that the KPI reads the chart's beliefs, the two select the same events, so a window overlapping a fourth day counts it. Signed-off-by: F.N. Claessen <felix@seita.nl>
…lexmeasures Kept all three changelog entries, dropping the duplication the two orderings produced. Signed-off-by: F.N. Claessen <felix@seita.nl>
Context: - CI's flake8 caught F401: get_sensor_stats was still imported in assets.py after get_kpis stopped calling it. I had been skipping flake8 locally, which is exactly why this reached CI. Change: - Remove the import. The sensor stats endpoint still uses the function; the KPI endpoint no longer does. Signed-off-by: F.N. Claessen <felix@seita.nl>
|
@nhoening — your three comments are addressed in the threads, and your bug is real. It turned out to be two, from one root. CI is green (13/13). What you foundReproduced exactly. On my machine the arbitrary winner was the other source — January 15 and 16 together reported 122 where you saw 100 — which is itself the diagnosis: there was no ordering at all.
downsample_value = dict(next(iter(sensor_stats.values())))["Sum over values"]So with January 15 computed and January 16 uploaded — different sources, exactly as you described — only one source was ever counted, and every range including the 16th reported that one. The second bug it uncoveredThose statistics carry no belief-time filter, so they aggregate every belief row. A revised value is therefore added to the value it revised. One day believed first as 50 and later as 122 reports 172, where the chart shows 122. The underlying problemBoth come from the same thing: the KPI aggregated raw belief rows, when the chart draws one value per event. A KPI is read beside the chart, so it should total what the chart totals.
One consequence worth your judgementThe KPI now inherits the chart's window semantics. Previously it selected on
|
Context: - @nhoening found that after the previous commit every one-day selection also counted its neighbour: with data on the 15th and 16th, the 15th reported 222 and the 14th reported 122. - Reading the chart's beliefs brought the chart's selection rule with it, which takes events *overlapping* the window. His sensor's daily events sit on the UTC grid while he reads them from +01:00, so every local day runs 23:00 to 23:00 UTC and straddles two events, and each event fell into two adjacent days. - Reproduced only once the events were placed on the UTC grid; with events at local midnight the totals were right, which is what made the first attempt look correct. Change: - Keep reading the beliefs the chart draws, but count each event under the window it starts in. Summing what the chart draws would count a straddling event under both neighbours. - This is a deliberate difference from the chart: the chart draws such an event in both days, the KPI counts it once. Signed-off-by: F.N. Claessen <felix@seita.nl>
|
Reproduced and fixed in 6ddda9f — thank you, that was a regression I introduced in the previous round. What went wrongReading the chart's beliefs brought the chart's selection rule with it. Your sensor's daily events sit on the UTC grid while you read them from +01:00, so every local day runs 23:00 → 23:00 UTC and straddles two events. Each event therefore fell into two adjacent local days and was counted under both. That is exactly the 122 / 222 / 100 you saw. Worth saying: my first attempt to reproduce it passed. I had placed the events at local midnight, where nothing straddles anything. Only once I put them on the UTC grid did your numbers appear — which is also what pins down what is particular about your data. The fixThe KPI still reads the beliefs the chart draws, but now counts each event under the window it starts in: This is a deliberate, narrow difference from the chart: the chart draws a straddling event in both days, the KPI counts it once. So "the KPI totals exactly what the chart draws" — which I asserted last round — was the wrong invariant, and I have corrected that test to compare only the events a window owns. Tests
flake8 and black clean, both at the versions pinned here. CI running. |
|
@nhoening Actually, what is your expectation to see as KPIs if you have daily sensor data in UTC and you view from, say, Tokyo? |
|
Thanks, I can confirm the behavior makes sense now!
Good question. You mean if the KPI data has no offset? I guess I would like a translation to my local timezone. But this isn't easy to adjust and/or to explain. I would like KPIs to be calculated on the same offset as input sensors (mostly the power sensors that matter), in the first place. But of course we cannot assume that. Or should reporting make that possible so KPIs make sense? It seems to me this is well worth of its own PR? Maybe a Claude design session first, to create an issue? |
The v1.0.1 patch release backported #2434, #2461, #2465, #2473, #2475 (partial - only the ad98460751d9 migration fix), #2479, #2480, #2482, #2499, #2502 and #2507 from this history to the 1.0.x branch. Move their changelog bullets out of the v1.1.0 section into a new v1.0.1 section, so the same fix is not listed as unreleased once 1.0.1 ships; the CLI changelog gets a matching 'since v1.0.1' section. The API change log keeps its own independent v3.0-NN numbering and is untouched. Signed-off-by: F.N. Claessen <claessen@seita.nl>
Description
Closes #2453.
KPIs on the asset page cover one day more than the chart beside them, and, as found in review, do not total what the chart draws at all when a sensor has more than one source or a revised belief.
ui/graphs: pass the chart's end date to the KPI endpoint unchangedapi/v3_0: total the beliefs the chart draws, rather than belief rows grouped by data sourcedocumentation/changelog.rstgetAssetKPIsadvanced its end date by a day, to "make the end date inclusive". But both callers already hand it the chart's exclusive end — the picker applies+1 daybefore calling, andevent_ends_beforeis exclusive on the initial page load. The endpoint ends its own window beforeendas well:_get_sensor_statsfiltersTimedBelief.event_start < end_dt.Advancing the date also mutated it in place, and that
Dateobject is the one held bystoreEndDateandpreviousResult.end, so the chart's own end date moved a day forward on every selection. Not advancing it at all fixes both, which is why this is a one-line change.Closes #2453
Where the extra day came from
The advance was not arbitrary, and it was once correct. Until #1909 this function took no arguments and read the picker directly:
picker.getEndDate()is the last day selected — 7 January for a 1–7 January selection — so advancing it produced the exclusive end the endpoint wants.#1909 ("fix: pass start and end dates to getAssetKPIs() on graphs page", closing #1908, shipped in v0.30.3, January 2026) gave the function
startDate/endDateparameters, because reading the picker broke page load. Both callers pass the chart's already-exclusive end: the picker handler applies+1 daybefore calling, andevent_ends_beforeis exclusive on the initial load. The advance inside was carried over unchanged, and from that point added a day too many.So this is a regression from #1909 rather than the undoing of a deliberate decision. Its own changelog entry stays as it is: #1909 shipped in v0.30.3, so this is a fix to released behaviour, not to an unreleased regression. I have left a comment where the advance used to be, so it does not get put back.
Warning
This changes reported KPI values. They were too high (or too wide, for min/max) by one day's worth of data. Anything tracking them will step down when this lands. The old numbers were wrong, but the change is visible.
Look & Feel
On a seeded asset carrying one unit of energy per day, asking for a three-day window:
endsentT0+3d(what the chart shows)T0+4d(what the UI sent)How to test
test_kpi_window_end_is_exclusiveinflexmeasures/api/v3_0/tests/test_assets_api.pyasserts that a three-day window over one daily value per day totals three, not four.To be explicit about what that test does and does not cover: it pins the endpoint's window semantics, which the page depends on. It does not exercise the JavaScript change, and would pass without it — there is no JavaScript test runner in this repo. The front-end half was verified by hand, by reading the
endparameter off the request the page issues.Manually: open an asset page with KPIs configured, select a range, and compare a KPI against the chart beside it. Before this change a sum covers one day more than the chart draws.
Notes
Split out of #2433 at the maintainer's request, because it changes user-visible numbers while that PR is a performance change. #2433 also carries this fix, so the two converge:
graphs.htmlmerges cleanly in either order (verified by a trial merge). The changelog entry lives here, and has been removed from #2433 to avoid a duplicate line.